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Summary 


Computer  models  continue  to  be  developed  and  modified  to  provide  more  accurate  weather 
forecasts.  The  Army  is  particularly  concerned  with  the  ability  of  forecast  models  to  provide 
appropriate  weather  data  for  regional  scales  including  complex  terrain.  This  study  investigated 
the  perfonnance  of  two  models  run  over  the  widely  varying  topography  of  northern  Utah. 
Twenty-four  hourly  forecasts  were  generated  daily  for  most  days  for  two  winter  and  two  summer 
months.  One  of  the  models  used  was  the  Pennsylvania  State  University/National  Center  for 
Atmospheric  Research  Mesoscale  Model  Version  5  (MM5),  which  was  run  in  a  triple  nest.  The 
other  model  used  was  a  non-nested  version  of  the  Weather  Research  and  Forecast  (WRF)  model. 
The  model  forecasts  for  surface  values  of  temperature,  dew-point  temperature,  and  wind 
components  were  compared  to  surface  observations  from  various  observation  sites.  The  average 
errors  found  through  these  comparisons  were  typically  higher  than  desired,  particularly  for  dew¬ 
point  temperatures,  as  documented  in  this  report.  Additional  analyses  will  be  perfonned  on  some 
specific  station  results  from  the  data  obtained  in  this  study,  but  future  evaluations  will  be  based 
on  a  more  recent  version  of  the  WRF  model,  which  includes  a  nesting  capability. 
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1.  Introduction 


Atmospheric  scientists  may  choose  from  a  variety  of  numerical  weather  prediction  computer 
models.  The  Pennsylvania  State  University/National  Center  for  Atmospheric  Research  (NCAR) 
Mesoscale  Model  Version  5  (MM5)  provides  regional-scale  forecasts  for  both  military  and 
civilian  users  (1 ).  A  new  model,  called  the  Weather  Research  and  Forecast  (WRF),  is  being 
developed  by  NCAR,  the  National  Oceanic  and  Atmospheric  Administration,  and  several 
universities  (2).  The  WRF  is  meant  to  improve  on  current  regional  forecasting  capabilities,  as 
well  as  to  provide  a  tool  useful  to  both  operational  weather  forecasters  and  researchers. 

Previous  studies  examined  MM5  performance  over  Utah  during  December  2002  through 
February  2003  (3,  4).  In  order  to  become  familiar  with  running  the  first  version  of  the  WRF,  it 
was  run  over  the  same  area  for  most  dates  in  July  and  August  2003  and  January  and  February 
2004.  The  MM5  was  also  run  for  the  same  location  and  dates.  However,  this  study  is  not  meant 
to  be  a  comparison  between  the  two  models.  The  version  of  WRF  used  did  not  provide  nesting 
capabilities,  while  the  MM5  output  was  from  the  inner  nest  of  three  nests. 

This  report  provides  a  preliminary  summary  of  the  average  errors  calculated  by  comparing  the 
forecast  models’  output  of  basic  parameters  with  the  surface  observations  available  from  the 
University  of  Utah  MesoWest  Cooperative  (5).  The  WRF  temperature,  humidity,  and  wind 
forecast  errors,  in  general,  were  quite  large.  This  was  also  true  for  the  MM5  forecasts,  which 
showed  average  temperature  and  dew-point  temperature  errors  significantly  larger  than  those 
found  in  the  previous  year’s  study. 


2.  Methodology 


Both  the  MM5  and  WRF  models  were  initialized  using  the  Global  Forecast  System  (GFS)  model 
(6).  Model  runs  were  initialized  at  1800  Zulu  Time  for  28  days  in  July  2003,  26  days  in  August 
2003,  28  days  in  January  2004,  and  29  days  in  February  2004.  The  models  provided  hourly 
output  covering  a  30-hour  period,  of  which  the  initial  6  hours  were  used  for  spin-up  purposes  and 
the  remaining  24  hours  were  used  for  validations. 

The  model  domains  incorporated  complex  terrain  in  northern  Utah  (fig.  1).  The  MM5  runs 
consisted  of  an  outer  nest  based  on  45-km  grid  point  spacing,  a  middle  nest  with  15-km  spacing, 
and  an  inner  nest  with  5-km  spacing.  The  inner  nest  contained  85-by-85  grid  points  covering  a 
420  km-by-420  km  area,  and  only  these  results  were  used  for  the  MM5  validations.  The  WRF 
runs  were  perfonned  without  nesting,  since  the  initial  version  of  the  WRF  did  not  provide  this 
capability.  A  larger  area  was  selected  to  provide  the  WFR  output,  consisting  of  a  5-km  grid 
point  spacing  for  101  -by- 101  grid  points  covering  500  km  by  500  km.  The  stations  used  for 
WRF  validations  were  the  same  as  the  ones  used  for  the  MM5  validations.  The  larger  WRF  area 
was  chosen  so  that  no  validations  would  need  to  be  performed  near  the  boundaries  of  the  WRF 
domain,  where  the  shortcomings  due  to  not  having  a  nested  model  would  be  expected  to  be 
greatest. 
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Figure  1.  The  MM5  model  domain  (inner  box  in  blue)  and  die  WRF 
model  domain  (outer  box  in  purple). 


For  each  model,  forecasts  of  temperature,  dew-point  temperature,  and  u-  and  v-wind  components 
were  interpolated  to  surface  station  locations.  These  forecast  values  were  then  compared  to 
hourly  observations  provided  through  the  Utah  MesoWest  Cooperative.  Approximately  40  to  70 
stations  were  available  for  most  hours.  The  observed  data  were  transferred  without  the  data  flags 
highlighting  questionable  observations,  and  the  station  location  density  was  not  uniform,  with 
many  stations  located  in  the  Salt  Lake  City,  UT,  vicinity.  Therefore,  the  average  error  statistics 
presented  in  the  next  section  may  be  skewed  by  bad  observations  and  by  a  particularly  good  or 
bad  forecast  being  multiplied  by  several  stations  in  the  same  vicinity. 


3.  Results 


In  general  the  average  forecast  errors  found  in  this  study  are  significantly  higher  than  expected. 
These  poor  results  cannot  be  solely  attributed  to  the  non-nested  version  1 .3  of  the  WRF  model, 
since  the  MM5  average  forecast  errors  were  almost  as  high,  or  even  higher,  than  the  WRF  errors 
The  MM5  errors  found  in  the  2004  winter  months  were  substantially  higher  than  those  seen  in  a 
previous  study  based  on  2003  winter  months.  That  study  did  not  include  any  model  runs  for  the 
summer  months.  The  following  sections  document  the  overall  results  for  the  basic  weather 
parameters. 

3.1  Temperature  Forecast  Errors 

The  average  temperature  forecast  errors  are  summarized  in  table  1 . 
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Table  1.  Average  temperature  forecast  errors. 


MM5 

MM5 

WRF 

Winter 

2003 

Winter 

2004 

Summer 

2003 

Winter 

2004 

Summer 

2003 

Mean  Error  (°C) 

1.3 

3.2 

0.1 

2.9 

-1.0 

Absolute  Error  (°C) 

2.5 

3.7 

2.9 

3.6 

2.8 

RMSE  (°C) 

3.4 

5.3 

3.7 

4.9 

3.6 

Correlation  Coefficient 

.83 

.69 

.86 

.72 

.91 

Number  of  Points 

58000 

54000 

72000 

55000 

75000 

NOTE:  RMSE  =  root  mean  square  error. 


The  winter  temperature  forecasts  displayed  a  warm  bias  around  3  °C  in  both  the  MM5  and  the 
WRF  runs.  The  summer  results  were  better,  with  very  little  bias  in  the  MM5  forecasts  and  a 
1  °C  cold  bias  in  the  WRF  forecasts.  The  absolute  error  amounts  were  very  similar  for  both 
models:  around  3.5  °C  in  the  winter  and  3.0  °C  in  the  summer. 


Figure  2  shows  the  variation  in  the  WRF  results  between  the  two  months  used  in  each  seasonal 
summary. 
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Figure  2.  WRF  temperature  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  month. 
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Averaging  errors  over  many  data  points  conceals  a  great  deal  of  variation  associated  with 
particular  times  and  places,  but  the  amount  of  data  involved  necessitates  summarization. 

Another  way  to  look  at  the  forecast  errors  is  by  forecast  time.  The  WRF  temperature  errors, 
averaged  for  the  winter  model  runs  by  forecast  hour,  are  shown  in  figure  3.  Forecast  hour  0  is  at 
5  p.m.,  local  Mountain  Standard  Time.  The  temperature  errors  are  unifonnly  around  3.5  °C  for 
hours  0  through  15,  and  then  climb  to  4.5  °C  by  hour  21,  before  decreasing  again  in  hours  22  and 
23.  Looking  at  the  average  temperature  errors  by  hour,  averaged  for  the  eight  individual  weeks 
included  in  the  winter  data,  all  but  two  of  the  weeks  showed  this  increase  in  forecast  error  in  the 
later  hours.  A  similar  trend  is  also  seen  in  the  MM5  forecasts. 
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Figure  3.  WRF  temperature  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  forecast  hour  for  the  winter  model  runs. 


The  WRF  temperature  errors  by  forecast  hour  for  the  summer  model  runs  did  not  exhibit  this 
maximum  error  at  hour  2 1  (see  fig.  4).  The  primary  difference  between  the  MM5  and  WRF 
temperature  errors  in  the  summer  months  is  highlighted  in  figure  5,  which  shows  the  MM5  bias 
switching  from  too  cold  during  the  daytime  hours  to  too  warm  during  the  night.  The  equivalent 
WRF  bias  is  also  too  cold  during  the  day,  but  stays  slightly  cold  during  the  night. 
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WRF  Temperature  Forecast  Errors 


o 

O) 

CD 

(D 

i_ 

O 

i_ 

LU 

0 

i_ 

3 

05 

s— 

0 

a 

£ 

0 


-0-  Mean  Error 
-D-Abs  Error 
-96-  Corr  Coef 


Summer  months;  by  forecast  hour 


Figure  4.  WRF  temperature  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  forecast  hour  for  the  summer  model  runs. 


MM5  Temperature  Forecast  Errors 


Summer  months;  by  forecast  hour 


Figure  5.  MM5  temperature  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  forecast  hour  for  the  summer  model  runs. 
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3.2  Dew-Point  Temperature  Forecast  Errors 

The  average  dew-point  temperature  forecast  errors  are  summarized  in  table  2. 


Table  2.  Average  dew-point  temperature  errors. 


MM5 

MM5 

WRF 

Winter 

2003 

Winter 

2004 

Summer 

2003 

Winter 

2004 

Summer 

2003 

Mean  Error  (°C) 

1.4 

2.5 

7.4 

4.1 

7.9 

Absolute  Error  (°C) 

2.9 

3.4 

7.5 

4.5 

8.1 

RMSE  (°C) 

4.2 

5.2 

9.4 

6.2 

10.0 

Correlation  Coefficient 

.73 

.69 

.50 

.66 

.42 

Number  of  Points 

58000 

54000 

72000 

53000 

71000 

The  seasonal  averages  again  obscure  the  differences  between  the  two  months  in  each  season,  as 
seen  in  figure  6. 
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Figure  6.  WRF  dew-point  temperature  errors  (left  axis)  and  correlation  coefficients  (right 
axis)  by  month. 


The  average  dew-point  temperature  errors  by  forecast  hour  for  the  winter  runs  decreased  in  hours 
17-23,  when  the  temperature  errors  had  increased  (fig.  7).  The  summer  runs  included  huge 
errors  in  dew-point  temperature  forecasts,  with  a  maximum  average  error  of  12  °C  at  forecast 
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hour  2  (fig.  8).  The  MM5  model  runs  reflected  similar  trends,  but  not  to  the  extremes  produced 
by  the  WRF. 


WRF  Dew-Point  Temperature  Forecast  Errors 
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Figure  7.  WRF  dew-point  temperature  forecast  errors  (left  axis)  and  correlation  coefficients 
(right  axis)  by  forecast  hour  for  the  winter  model  runs. 


WRF  Dew-Point  Temperature  Forecast  Errors 


Summer  months;  by  forecast  hour 


Figure  8.  WRF  dew-point  temperature  forecast  errors  (left  axis)  and  correlation  coefficients 
(right  axis)  by  forecast  hour  for  the  summer  model  runs. 
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3.3  Relative  Humidity  Forecast  Errors 

The  average  relative  humidity  forecast  errors  are  summarized  in  table  3. 


Table  3.  Average  relative  humidity  errors. 


MM5 

MM5 

WRF 

Winter 

2003 

Winter 

2004 

Summer 

2003 

Winter 

2004 

Summer 

2003 

Mean  Error  (%) 

-1 

-5 

16 

5 

21 

Absolute  Error  (%) 

15 

17 

19 

16 

22 

RMSE  (%) 

19 

22 

23 

22 

27 

Correlation  Coefficient 

.49 

.43 

.63 

.31 

.68 

Number  of  Points 

58000 

54000 

72000 

55000 

75000 

As  can  be  deduced  from  the  substantial  warm  bias  in  the  dew-point  temperature  forecasts,  the 
WRF  relative  humidity  forecasts  are  usually  too  high.  The  exception  is  forecast  hours  17-23  in 
the  winter  months,  which  show  a  bias  for  the  relative  humidity  forecasts  to  be  between  0  and 
10  °C  too  low.  The  winter  MM5  forecasts  show  this  slight  low  bias  throughout  all  the  forecast 
hours,  with  a  somewhat  stronger  high  bias  for  the  summer  forecast  hours. 

3.4  Wind  Component  Forecast  Errors 

The  average  «-wind  component  forecast  errors  are  summarized  in  table  4. 

Table  4.  Average  zr-wind  component  errors. 


MM5 

MM5 

WRF 

Winter 

Winter 

Summer 

Winter 

Summer 

2003 

2004 

2003 

2004 

2003 

Mean  Error  (m/s) 

0.3 

0.5 

0.1 

0.6 

0.1 

Absolute  Error(m/s) 

1.8 

1.6 

2.2 

1.6 

1.8 

RMSE  (m/s) 

2.7 

2.4 

3.1 

2.5 

2.6 

Correlation  Coefficient 

.32 

.36 

.20 

.36 

.29 

Number  of  Points 

51000 

42000 

63000 

42000 

63000 
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The  average  v-wind  component  forecast  errors  are  summarized  in  table  5. 


Table  5.  Average  v-wind  component  errors. 


MM5 

MM5 

WRF 

Winter 

Winter 

Summer 

Winter 

Summer 

2003 

2004 

2003 

2004 

2003 

Mean  Error  (m/s) 

0.1 

0.7 

0.9 

0.4 

0.4 

Absolute  Error(m/s) 

2.0 

2.3 

2.5 

2.1 

2.0 

RMSE  (m/s) 

2.8 

3.2 

3.3 

3.0 

2.8 

Correlation  Coefficient 

.52 

.44 

.42 

.47 

.51 

Number  of  Points 

51000 

42000 

63000 

42000 

63000 

It  is  not  unusual  for  wind  component  correlation  coefficients  to  be  less  than  0.5.  The  northern 
Utah  area  used  for  these  forecasts  contains  large  topographical  variations  over  short  horizontal 
distances,  resulting  in  very  localized  wind  flow  effects  that  were  difficult  for  the  WRF  and  MM5 
regional  models  to  replicate. 

3.5  Wind  Speed  Forecast  Errors 

The  average  wind  speed  forecast  errors  are  summarized  in  table  6. 

Table  6.  Average  wind  speed  errors. 


MM5 

MM5 

WRF 

Winter 

2003 

Winter 

2004 

Summer 

2003 

Winter 

2004 

Summer 

2003 

Mean  Error  (m/s) 

0.3 

1.3 

0.6 

1.1 

-0.1 

Absolute  Error  (m/s) 

1.8 

2.0 

2.1 

2.0 

1.8 

RMSE  (m/s) 

2.4 

2.7 

2.7 

2.7 

2.3 

Correlation  Coefficient 

.44 

.41 

.24 

.41 

.30 

Number  of  Points 

52000 

43000 

64000 

43000 

65000 

The  average  wind  speed  absolute  errors  are  close  to  2  m/s  for  both  models  and  both  seasons. 
Although  this  error  is  not  particularly  large,  the  correlation  coefficients  don’t  indicate  a 
significant  amount  of  skill.  The  average  monthly  error  did  not  vary  for  the  two  months  within 
each  seasonal  category.  The  WRF  forecasts  for  the  winter  months  showed  a  higher  wind  speed 
bias  and  absolute  error  in  the  later  forecast  hours  (fig.  9).  On  the  other  hand,  the  WRF  forecasts 
for  the  summer  months  included  a  low  bias  for  hours  0-5  and  17-23  (fig.  10).  The  MM5  wind 
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speed  forecasts  for  the  summer  months  also  had  a  low  bias  for  hours  18-23,  but  demonstrated  a 
pronounced  high  bias  during  the  nighttime  forecast  hours  3-15  (fig.  1 1). 
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Figure  9.  WRF  wind  speed  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  forecast  hour  for  the  winter  model  runs. 
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Summer  months;  by  forecast  hour 


Figure  10.  WRF  wind  speed  forecast  errors  (left  axis)  and  correlation  coefficients  (right  axis) 
by  forecast  hour  for  the  summer  model  runs. 
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MM5  Wind  Speed  Forecast  Errors 


Summer  months;  by  forecast  hour 


Figure  11.  MM5  wind  speed  forecast  errors  (left  axis)  and  correlation  coefficients  (right 
axis)  by  forecast  hour  for  the  summer  model  runs. 


3.6  Wind  Direction  Forecast  Errors 

The  average  wind  direction  forecast  errors  are  summarized  in  table  7. 
Table  7.  Average  wind  direction  errors. 


MM5 

MM5 

WRF 

Winter 

Winter 

Summer 

Winter 

Summer 

2003 

2004 

2003 

2004 

2003 

Absolute  Error  (°) 

51 

49 

58 

47 

51 

RMS  Vector  Error  (m/s) 

3.9 

3.9 

4.5 

3.9 

3.8 

Number  of  Points 

51000 

42000 

63000 

42000 

63000 

NOTE:  RMS  =  root  mean  square 

The  average  wind  direction  errors  generally  fell  between  50°  and  60°,  with  the  MM5  winter 
errors  equivalent  to  those  from  the  previous  winter.  Slightly  higher  errors  occurred  in  the  MM5 
forecasts,  particularly  in  the  early  forecast  hours  valid  during  the  evening  and  nighttime.  The 
July  MM5  forecasts  contained  the  highest  wind  direction  errors;  however,  there  was  little 
difference  in  WRF’s  performance  for  the  individual  months.  The  RMS  vector  errors  did  not 
exhibit  much  variation  by  forecast  hour  or  season. 
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4.  Conclusions 


This  report  documents  the  average  MM5  and  WRF  forecast  errors  for  basic  surface  parameters 
found  over  1 1 1  days  (in  July  and  August  2003  and  January  and  February  2004)  over  northern 
Utah.  The  approximate  ranges  of  absolute  error  values  averaged  over  the  two  months  within 
each  season  are  summarized  below: 

•  Temperature  forecast  errors:  3. 0-3. 5  °C 

•  Dew-point  temperature  forecast  errors:  3. 5-8.0  °C 

•  Relative  humidity  forecast  errors:  15-20  percent 

•  Wind  speed  forecast  errors:  2.0  m/s 

•  Wind  direction  forecast  errors:  50-60° 

These  errors  are  generally  higher  than  would  be  desired  in  an  operational  weather  forecast. 

Cases  with  very  low  wind  speeds  were  not  excluded  in  the  statistics,  which  resulted  in  lower 
average  wind  speed  forecast  errors,  but  also  contributed  to  higher  wind  direction  errors  based  on 
the  inclusion  of  light  and  variable  wind  situations. 

Some  additional  analyses  of  these  model  forecast  outputs  will  be  performed  to  consider  the 
model  results  at  specific  station  locations.  However,  the  primary  emphasis  will  shift  to  running 
and  evaluating  using  a  newer,  nested  version  of  the  WRF  model. 
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